Information inputting/specifying method and information inputting/specifying device

ABSTRACT

An object surface is imaged by an imaging unit ( 11 ) and, based on the imaging information by the imaging unit ( 11 ), the information corresponding to the object is read out and recognized to identify the object, as an object identification code indicating the information set corresponding to the object stored in the storage unit ( 14 ) and the information for specifying the information in the information set corresponding to the object are matched to a position on the object surface by an imaging unit ( 12 ) and a controller ( 13 ). In this manner, the knowledge stored in the information system can be freely accessed without producing break points, if the knowledge is to be used.

TECHNICAL FIELD

This invention relates to information inputting and specifying methodsand apparatus employing picture recognition.

BACKGROUND ART

As a device which, by combining the functions of pre-existing AV (audiovisual) equipment, information equipment or communication equipment,enables the inputting/outputting of the multi-media information, thereare currently provided a variety of multi-media equipment, such as aportable information terminal, having a communication function, a VTR(video tape recorder) with a built-in camera, having the communicationfunction, or a personal computer loaded not only with the communicationfunctions but also with the AV functions. The construction of amulti-media system, in which a variety of multi-media equipment areinterconnected over a wired communication network, such as Ethernet, atoken ring or an ATM, or a wireless communication network, such asinfrared ray communication, PHS communication or satellitecommunication, is proceeding.

In the conventional multi-media equipment, the keyboard, mouse, touchpanel, dedicated controller or the microphone, used as input devices fora personal computer, are directly used as the inputting devices. Inthese inputting devices, all of the clues for information inputting andinstructions are in the computer, such that, in a boundary between theobjects lying in a sphere of human living environment and computer data,there exists a significant barrier or partition which tends to obstructcontinuous thought of the human being.

It may be currently estimated that the access to the Internet underutilization of a portable terminal represented by the iMode performs animportant role. In a portable information terminal, represented first ofall by a portable telephone set, a multi-function key and a jog key fortelephone are adopted in view of limitations on the space occupied bythe terminal device and the operating constraint imposed by thetelephone set. Thus, for inputting a key word for WEB retrieval and forinputting site addresses, the user is required to be skilled in theoperation-of the multi-function key, which means considerabledifficulties especially for beginners.

That is, the operating methods for inputting devices, such as keyboards,mouses, touch panels, dedicated controllers or microphones, used asinputting devices for conventional multi-media equipment, cannot be saidto be matched to the senses of the human being. For users not versed inthe computer operations, the operating method cannot be envisionedintuitively, such that specialized knowledge and training are requiredfor comprehending the operating methods.

Additionally, the human being relies upon various objects lying in thethree-dimensional space as a clue for memorization or thinking, suchthat, when the user thinks in the three-dimensional living environmentor exploits the knowledge stored in the information system, it isdesirable that an information system inherently supporting theintellectual activities of the human being can be easily accessed in amanner not producing breaking points.

DISCLOSURE OF THE INVENTION

It is therefore an object of the present invention to provideinformation inputting and specifying methods and apparatus in which theknowledge stored in the information system can be freely accessedwithout producing break points.

According to the present invention, the objects lying in the user'sliving environment are associated with a set of the information storedin an information system, an object discriminating code indicating theinformation set corresponding to the objects and which is stored in astorage device of the information system is added to the surface of eachobject, so that the information for specifying the information in theinformation set corresponding to the object is matched to a position onthe object surface in such a manner that the information correspondingto the object is automatically read out and recognized to identify theobject and also in such a manner that the information set in the storagedevice and the information in the information set will be specified andselected by specifying the position on the object surface.

That is, by associating an object with the information set stored in theinformation system, it is possible for the human user to specify andselect the targeted information set, by indicating the object, withoutbecoming dissociated from his or her living environment. Additionally,the human user is able to specify and select the necessary informationin the information set as selected by specifying a position on theobject in a state close to a spontaneous state as if the information isprovided in his or her living environment.

The present invention provides an information inputting and specifyingmethod including matching an object identification code, indicating a,set of the information corresponding to the object, and the informationfor specifying the information in the information set corresponding tothe object, to the surface of each object, automatically reading out andrecognizing the information corresponding to an object for identifyingthe object, and specifying a position on the object surface to specifyand select the information set in the storage device and the informationin the information set.

The present invention also provides an information inputting andspecifying apparatus including storage means having stored therein a setof information corresponding to an object, object identification codereadout means for reading out an object identification code indicatingthe set of information corresponding to the object stored in the storagemeans, from the surface of said object, position information detectionmeans for detecting the information, specifying the information in theinformation set corresponding to the object, as the position informationon the surface of the object, and information processing means forrecognizing the information corresponding to the object and foridentifying the object, by matching the object identification code onthe surface of the object specifying the information set correspondingto the object, stored in the storage means, and the informationspecifying the information in the information set corresponding to theobject, to a position on the object surface, based on the objectidentification code read out by the object identification code readoutmeans and on the position information read out by the positioninformation detection means. The information set in the storage meansand the information in the information set are specified and selected byspecifying and selecting a position on the object surface.

In the information inputting and specifying apparatus according to thepresent invention, imaging means for imaging the object surface isprovided as the object identification code readout means and as theposition information detection means. The information processing meansreads out and recognizes the information corresponding to an object, bymatching an object identification code, indicating a set of theinformation corresponding to an object, stored in the storage means, andthe information for specifying the information in the information setcorresponding to the object, to the surface of the object, foridentifying the object.

In the information inputting and specifying apparatus, according to thepresent invention, the object is a sheet-shaped information inputtingmedium.

In the information inputting and specifying apparatus according to thepresent invention, sheet-shaped coordinate inputting means is providedas the position information detecting means. The sheet-shaped coordinateinputting means detects the position information on the sheet-shapedinformation inputting medium.

In the information inputting and specifying apparatus, according to thepresent invention, the sheet-shaped coordinate inputting means is formedof a transparent material.

In the information inputting and specifying apparatus according to thepresent invention, according to the present invention, the sheet-shapedcoordinate inputting means is meshed.

In the information inputting and specifying apparatus, according to thepresent invention, optical information readout means for reading out theobject identification code optically readably recorded on the objectsurface is provided as the object identification code readout means.

In the information inputting and specifying apparatus according to thepresent invention, magnetic information readout means for reading outthe object identification code magnetically readably recorded on theobject surface is provided as the object identification code readoutmeans.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing a functional structure of aninformation inputting and specifying apparatus according to the presentinvention.

FIG. 2 is a schematic perspective view showing the information inputtingand specifying apparatus.

FIG. 3 is a schematic plan view of an information inputting sheet usedin the information inputting and specifying apparatus.

FIG. 4 is a plan view showing the structure of a Umark figure codeindicated in the information inputting sheet.

FIGS. 5A and 5B show typical code forms of the Umark figure code

FIG. 6 shows typical arrays of the Umark figure codes.

FIG. 7 is a block diagram showing a hardware structure of essentialportions of the information inputting and specifying apparatus.

FIG. 8 schematically shows the definition of an architecture of aletter/character inputting interface operating as an user interface byvisual interaction at a portable terminal.

FIG. 9 is a flowchart showing a processing sequence of letter/characterrecognition by a pattern matching method.

FIG. 10 shows an illustrative structure of a data servicing systememploying a portable terminal having a figure code recognition programinstalled therein.

FIG. 11 shows an illustrative structure of a multi-media data servicingsystem employing a portable terminal having the figure code recognitionprogram installed therein.

FIG. 12 shows an illustrative structure of a electrical householdequipment control servicing system employing a portable terminal havingthe figure code recognition program installed therein.

FIG. 13 shows an illustrative structure of a mail servicing systememploying a portable terminal having the figure code recognition programinstalled therein.

FIG. 14 is a perspective view showing the overall configuration of aviewing tablet system.

FIGS. 15A and 15B are schematic perspective views showing acolumnar-shaped object and a triangular pyramid-shaped object, as pickedup by an imaging unit of the information inputting specifying device.

FIG. 16 is a schematic plan view of the information inputting specifyingdevice having an optical information readout unit and a sheet-likecoordinate inputting unit.

FIG. 17 is a schematic front view showing an electromagnetic elementforming a basic unit of a magnetic information readout unit.

FIG. 18 is a schematic perspective view of a number of magneticinformation readout units comprised of a two-dimensional matrixarrangement of a number of the electromagnetic elements as basic units.

BEST MODE FOR CARRYING OUT THE INVENTION

Referring to the drawings, certain preferred embodiments of the presentinvention are explained in detail.

The present invention is directed to an information inputting andspecifying device 10, shown for example in FIG. 1. The informationinputting and specifying device 10, shown in FIG. 1, includes an imagingunit 11, an image recognition unit 12, a controller 13, a storage unit14 and an outputting unit 15. The controller 13 is connected viacommunication networks 18 and 19, such as a network or a home bus, to aninformation distributing unit 16 and to an external device 17.

The imaging unit 11 images a recognition object pattern, added to asurface of an object, such as paper sheet, and an access instructionpattern for accessing by for example a finger by a user. Specifically,the imaging unit is comprised of a CCD viewer camera 21, mounted on aviewer 25A, as shown for example in FIG. 2. The CCD viewer camera 21images an information inputting sheet 29, as set on a media base 28, towhich the viewer 25A is mounted upright.

The information inputting sheet 29, shown in FIG. 3, is used forcontrolling the equipment, and includes a display unit 29A forrecognition codes for remote control, provided from one equipment typeto another or from one equipment to another. The information inputtingsheet 29 also includes a display unit 29B of control contents,associated with the equipment type or with the equipment, and a displayunit 29C for control stop commands for control contents associated withthe equipment type or with the equipment.

In the display unit 29A for the recognition codes, there is displayed arecognition code 30, referred to below as a Umark-figure code 30,comprised of a circle mark 31 and eight rectangular marks 32, arrangedradially therearound for extending in the eight directions, forming aunit of rotational figure elements, as shown in FIG. 4. One unit of theUmark figure code 30 defines eight codes associated with the eightrectangular marks 32.

The Umark figure code 30 is adapted to be used on being selectivelydaubed in order to permit free definition by the user. Since the Umarkfigure code is provided with a code detection mark 40, picturerecognition may be easily achieved from the code taken alone.

Since one unit of the Umark figure code 30 depends on the figurerotating direction, the base-8 notation, shown in FIG. 5A, is to be thebasic notation. However, decimal notation may also be used in order topermit more facilitated recognition by the user.

In actually using the Umark figure code, plural units of the Umarkfigure code 30 may be combined and arrayed in one direction, that is inthe horizontal direction or in the vertical direction, as shown forcases 1 and 2 in FIG. 6. Alternatively, high efficiency coderepresentation, exploiting the blocked arrangements, as shown for thecases 3 to 5 shown in FIG. 6, may be used. The image recognition of theUmark figure code 30 may be achieved readily with the code detectionmark 40 as reference.

The image recognition unit 12 recognizes the image information, aspicked up by the imaging unit 11. The image recognition unit is suppliedwith the image information, obtained as the imaged output of theinformation inputting sheet 29 by the CCD viewer camera 21 shown in FIG.2. The image recognition unit 12 discriminates and authenticates anobject for recognition for the image information obtained as an imagingoutput of the information inputting sheet 29 by the CCD viewer camera 21and outputs the results of discrimination and authentication to thecontroller 13. Following correlation processing with the informationcorresponding to the results of recognition of the objects forrecognition, as later explained, the image recognition unit 12incidentally recognizes the user input/selection command to output theresults of recognition to the controller 13.

The controller 13 controls the information transmission/reception amongrespective components as follows:

First, the controller checks whether or not the results of recognitionof the object for recognition by the image recognition unit 12 may becorrelated with the information stored in the storage unit 14. If thecorrelation with the information held in the storage unit 14 ispossible, such correlation with the information held in the storage unit14 is effectuated. That is, if the correlation of the results ofrecognition with the information stored in the own device is possible,such correlation is effectuated. If the correlation with the informationheld in the storage unit 14 is not possible, the results of recognitionof the object for recognition by the image recognition unit 12 are sentto the information distributing unit 16 to receive the results ofcorrelation from the information distributing unit 16. In this case, thecorrelated information is stored in the information distributing unit 16or in an information processing apparatus present on an externalcommunication network connected to the information distributing unit 16.After the results of recognition of the object for recognition by theimage recognition unit 12 have been correlated with the correspondinginformation, the controller 13 reads in the correlated information intoa transient storage unit provided in the controller 13. The informationso read in includes, for example, the display information output to anddemonstrated in the outputting unit 15, the control information forcontrolling the external device 17, the link information to the otherinformation, and the constituent information of the informationinputting sheet 29. For each subsequent inputting/selecting command fromthe image recognition unit 12, the authentication processing isperformed, the display image is updated and control on the externaldevice 17 is exercised to display the operating contents and the resultsof information distribution on the outputting unit 15.

The various information read into the transient storage unit of thecontroller 13 are hereinafter explained.

The display information includes for example the letters or characters,images, layout information and the sound information. Based on this;display information, the controller 13 controls the display on thescreen of the outputting unit 15 and the utterance.

The control information includes control commands for controlling theexternal device 17 when an inputting/instructions are made from theuser. The controller 13 controls the external device 17 based on thiscontrol information. For example, if the external device 17 is the VTR(video tape recorder), the control information may include the replaycontrol information, stop control information, fast feed controlinformation, rewind control information and the pause controlinformation, for controlling the VTR operation.

The link information includes addresses for reading out the otherinformation stored in the storage unit 14, information distributing unit16 and in the external information processing apparatus connected to acommunication network. The controller 13 performs the processing ofreading out the information based on this link information. For example,the URL (Uniform Resource Locator), used in the Internet, may be used asthis link information.

The constitution information includes the information for representingwhat is recorded in the display unit 29B of the information inputtingsheet 29 having control contents corresponding to the equipment or theequipment type. Specifically, the constitution information includes thephysical positions on the sheet as indicated by the user making theinput/selection instructions, and the next processing commandsassociated with the physical positions. For example, if the positioncoordinates and the positions are indicated by the user, these positioncoordinates or the positions are stated in association with theprocessing commands for executing the aforementioned control informationor the link information.

The storage unit 14 memorizes the information correlated with thepattern of the object for recognition. The information stored in thestorage unit 14 is the information for controlling the external device17. These information are transferred to the storage unit 14 using theexternal information processing apparatus connected to the network viainformation distributing unit 16 in advance or using satellitecommunication.

The outputting unit 15 is made up by operating contents by theinput/output instructions from the image recognition unit 12, and adisplay, a loudspeaker or a printer for outputting the results of theinformation distribution.

The information distributing unit 16 is connected to the controller 13over a wired communication network, such as Ethernet, a token ring orATM, or a wireless communication network, such as infrared raycommunication, PHS communication or satellite communication. If thecorrelation by the controller 13 between the results of recognition ofthe object for recognition by the image recognition unit 12 and theinformation stored in the storage unit 14 has not been achieved withsuccess, the information distributing unit 16 proceeds to the processingof correlating the results of recognition of the object for recognitionand the information stored in the external information processingapparatus connected to the network.

The external device 17 is an equipment controlled by the controller 13responsive to the inputting/selecting instructions, and is connected tothe controller 13 by the wired or wireless communication line.

Specifically, with the information inputting and specifying device 10 ofthe above-described configuration, the display unit 29A of theequipment-based or equipment type-based remote controlling codes forrecognition of the information inputting sheet 29 is read, while theinformation corresponding to the object may be automatically read outand recognized to identify the object, as the object identification codeon the surface of each object indicating the set of the informationcorresponding to the object and which is stored in a storage system inan information system, and the information for specifying theinformation in the information set corresponding to the object, arematched to a position of the object surface, to identify the object. Theinformation set in the storage unit and the information in theinformation set can be specified and selected by specifying the positionon the display unit 29B of the information inputting sheet 29 having thecontrol content corresponding to the equipment type or equipment.

Referring to FIG. 7, the information inputting and specifying device 10is made up by a CPU (Central Processing Unit) 51, an image processing,unit 52, a camera interface 53, a display interface 54, a networkinterface 55, a home network interface 56, a ROM (Read-Only Memory) 57,a RAM (Random-Access Memory) 58, an HDD (Hard Disc Device) 59 and a mainbus 60.

The CPU 51 exercises controls based on a system program stored in theROM 57 or a program transferred to the RAM 58. The CPU 51 implements aportion of the functions of the image recognition unit 12 and a portionof the functions of the controller 13.

The image processing unit 52 is supplied through camera interface 53with an image picked up by the CCD viewer camera 21 to perform variousimage processing operations, such as the processing of imagerecognition. This image processing unit 52 partially implements thefunctions of the image recognition unit 12 of FIG. 1. The camerainterface 53 is supplied with an image picked up by the CCD viewercamera 21 and transforms the input image to an image complying with thesignal format which is in keeping with the image processing unit 52 tooutput the resulting image. The display interface 54 is supplied withthe display data processed by the image processing unit 52 andtransforms the data into signals which are in keeping with the viewer52A to output the resulting signals. The display interface 54 and theviewer 25A implement a portion of the functions of the outputting unit15 of FIG. 1.

The network interface 55 is an interface for connection to a serverstation. By this interface, connection may be made to an externalnetwork. The network interface 55 is a device partially implementing thefunctions of the information distributing unit 16 and the communicationnetwork 18 of FIG. 1.

The home network interface 56 is an interface for connection to theexternal device 17. The CPU 51 controls the external equipment throughthis home network interface 56. The home network interface 56 and theexternal equipment are devices partially implementing the functions ofthe external device 17 and the communication network 19.

In the ROM 57, there are stored programs or data necessary for the CPU51 to execute various processing operations. The RAM 58 transientlyholds programs or data necessary for the CPU 51 to execute variousprocessing operations. The ROM 57 and the RAM 58 implement portions ofthe functions of the controller 13 of FIG. 1.

The HDD 59 holds the information correlated to the information inputtingsheet 29. The HDD 59 is a device implementing the functions of thestorage unit 14 of FIG. 1.

The function of automatically reading out and recognizing theinformation corresponding to the object, as the object identificationcode on the surface of each object indicating the set of the informationcorresponding to the object and which is stored in the storage system inthe information system, and the information for specifying theinformation in the information set corresponding to the object, arematched to a position on the object surface for identifying the object,and of specifying the position on the display unit 29B of the controlcontents associated with the equipment type or the equipment to instructand select the information set in the storage unit and the informationin the information set, may be loaded on for example the portabletelephone set for operation as the user interface by visual interaction.

In this interface, (1) letter/character definition is by employing theletters or characters used in the printed matter or handwritten by theuser and (2) the letters or characters in need of letter/characterinputting are captured using a camera annexed to the portable telephoneset. Since the meaning and attributes of the letters or characterscaptured are multifarious, as for example WEB site addresses, letters orcharacters retrieved, or filenames, the processing to be performed isspecified in advance of inputting, such as by employing the figurecodes, such as the Umark figure codes in the information inputting sheet29.

That is, the architecture of the letter/character inputting interface isdefined as shown for example in FIG. 8.

In FIG. 8, (A) specifies figure codes for letter/character processing inhttp, ftp, telnet of file. In FIGS. 8, (B), (C) and (D) state forexample the node names of the URL, names of the organizations and typesof the organizations, respectively. FIG. 8(E) is an area stating thenation symbols.

That is, the letter/character processing codes are basically made up bythe code detection figures of FIG. 8(A)-1 and the Umark code figures ofFIG. 8(A)-2.

The code detection figures are used as an index for normalizing thefigure codes by extracting the figure codes specifying theletter/character processing attributes from a two-dimensional: imagepicked up by a camera and by carrying out the threshold value decisionor correction by rotation by a so-called P-tile method in which athreshold value T for binary coding is determined so that, when thegross area ratio in an image of the object is known, the proportion of apixel 1 in a bi-value image will be equal to P.

For the normalized figure codes, each figure code of FIG. 8(A)-2 isseparated and the center of gravity of each figure code is calculated byimage processing. Eight direction fitting is then performed from thecenter of gravity vector for conversion to eight sorts of digital data.

This figure code recognition program is built into the portabletelephone set so that the letter/character processing program will bebooted the instant the figure codes by the camera are recognized.

The pattern matching method, for example, is applied as a method forletter/character recognition for performing letter/character extractionusing the camera.

In the letter/character recognition by the pattern matching method,processing is carried out in the sequence of image data pre-processing(step S1), image data slicing (step S2), letter/character analysis andfeature extraction (step S3), letter/character pattern matching (stepS4) and letter/character decision and display (step S5).

By the portable telephone set, which has the figure code recognitionprogram installed therein, being used in increasing numbers, the dataservices as shown for example in FIG. 10 may be rendered.

That is, in a data servicing system 100, shown in FIG. 10, a provider110 pays licensing fee in a net bank 120 (A1).

The net bank 120 distributes a license to the provider 110 (A2).

The contents of a seal 120A of the URL address, pasted on a poster 130,are captured by a CCD viewer camera 140A of a portable telephone set 140(A3).

The portable telephone set 140 automatically transmits the so capturedURL address to a base station 150 (A4).

The base station 150 transmits the URL address to an exchange station160 of a carrier (A5).

The exchange station 160 of the carrier transmits the URL address to theprovider 110 via e.g., Internet (A6).

The provider 110 transmits the HTML (HDML) to the exchange station 160of the carrier (A7).

The exchange station 160 of the carrier trims the received HTML (HDML)to transmit the data to the base station 150 (A8).

The base station 150 transmits data to the portable telephone set 140(A9).

The net bank 120 tolls the access fee (A10).

The multi-media services may also be rendered, as shown for example inFIG. 11.

That is, in multi-media service system 200, shown in. FIG. 11, aprovider 210 pays the license fee to the net bank 220 (B1).

The net bank 220 distributes the license to the provider 210 (B2).

The contents of a seal 230A of the URL address, such as short movie(game or picture) pasted on the poster 230, are captured by a CCD viewercamera 240A of a portable telephone set 240 (B3).

The portable telephone set 240 automatically transmits the captured URLaddress to a base station 250 (B4).

The base station 250 transmits the URL address to a carrier exchangestation 260 (B5).

The carrier exchange station 260 transmits the URL address to theprovider 210 over e.g., the Internet (B6).

The provider 210 transmits the HTML (HDML) to the carrier exchangestation 260 (B7).

The carrier exchange station 260 trims the received HTML (HDML) totransmit the data to a relevant base station 250 (B8).

The base station 250 transmits a multi-media object to the portabletelephone set 240 (B9).

The net bank 220 tolls the access fee (B10).

Moreover, electrical household equipment control services, shown forexample in FIG. 12, may also be rendered.

That is, in an electrical household equipment control service system300, shown in FIG. 12, a provider 310 pays the license fee to a net bank320 (C1).

The net bank 320 distributes a license to the provider 310 (C2).

The contents of the electrical household equipment control information330A, pasted to a book (memo book) 330, are captured by a CCD viewercamera 340A of a portable telephone set 340 (C3).

The portable telephone set 340 automatically transmits the capturedelectrical household equipment control information to a base station 350(C4).

The base station 350 transmits the electrical household equipmentcontrol information to a carrier exchange station 360 (C5).

The carrier exchange station 360 authenticates the received electricalhousehold equipment control information and, if the information isauthenticated, the carrier exchange station transmits the electricalhousehold equipment control information to a relevant provider 310 (C6).

The provider 310 transmits the electrical household equipment controlinformation to a customer 370 (C7).

The customer 370 controls the relevant electrical household equipmentbased on the received electrical household equipment control information(C8).

An image of a security camera of the customer 370 is sent (C9).

The net bank 320 tolls the access fee (C10).

Additionally, the mail services shown for example in FIG. 13 may also berendered.

That is, in a mail servicing system 400, shown in FIG. 13, a provider410 pays the license fee to a net bank 420 (D1).

The net bank 420 distributes a license to the provider 410 (D2).

The contents of an E-mail address 430A, pasted to a poster 430, arecaptured by a CCD camera 440A of a portable telephone set 440 (D3).

If the speech is input, the speech is automatically transformed intoletters or characters to prepare an E-mail. The E-mail prepared may betransmitted to a base station 450 on pressing a transmission key of theportable telephone set 440 (D4).

The base station 450 transmits an E-mail to a carrier exchange station460 (D5).

The carrier exchange station 460 transmits an E-mail to a relevantcustomer address over e.g. the Internet (D6).

When the transmission is terminated as regularly, a message indicatingthe completion of transmission is transmitted to the carrier exchangestation 460 (D7).

The carrier exchange station 460 transmits the received completionmessage to the relevant base station 450 (D8).

The base station 450 transmits the transmission completion message tothe relevant portable telephone set 440 (D9).

The net bank 420 tolls the access fee (D10).

The function of automatically reading out and recognizing theinformation corresponding to the object for identifying the object, asan object identification code on the object surface indicating theinformation in the set of the information corresponding to the objectand which is stored in the storage system in the information system tothe surface of each object, and the information for specifying theinformation in the information set corresponding to the object, arematched to the position on the object surface, and the functionspecifying and the function of selecting the information set in thestorage unit, and the information in the information set, by specifyinga position on the object surface, may be exploited to construct aviewing tablet system 500 shown for example in FIG. 14.

This viewing tablet system 500 is made up by a loaded type wirelesscamera 510, that is able to capture an image at all times from a line ofsight position of the viewer, and an image recognition presentationdevice 520 dynamically comprehending the contents of operations andinstructions to the object in keeping with the movement of the field ofview of the user to furnish adequate information to the user.

In this viewing tablet system 500, the user loads the wireless camera510 on the line of sight position, so that, as the line of sight ismoved, the object may be automatically recognized to issue operationalinstructions within the field of view. That is, by employing the loadedtype wireless camera 510, the image of the object can be dynamicallygrasped from the viewing point position of the user.

In this viewing table system 500, the user sets a space code sheet at aposition on an object, such as book or paper sheet, added by the Umarkcategory set. The user then causes the Umark category code of the objectto be recognized by the loaded type wireless camera 510 within the fieldof view. When further the user touches a specified portion of the papersheet with a finger by way of giving an input and instructions, thefinger and the partial image of the paper sheet are input-from theloaded type wireless camera 510 to the image recognition presentationdevice 520. The image recognition presentation device 520 thenrecognizes this partial image as an image to verify the contents of theuser's instructions to furnish the optimum information.

According to the present invention, described above, an object lying inthe user's living environment is matched to a set of the informationstored in an information system, whilst an object discriminating codeindicating the information set matched to the object and which is storedin a storage device of the information system is added to the surface ofeach object, so that the information for specifying the information inthe information set corresponding to the object is matched to theposition on the object surface, in such a manner that the informationmatched to the object is automatically read out and recognized toidentify the object, and also in such a manner that the information setin the storage device and the information in the information set will bespecified and selected by specifying the position on the object surface.In this manner, the knowledge stored in an information system may befreely accessed and exploited in a manner not producing breaking points.

In the above-described information inputting and specifying device 10,the pattern of the object of recognition, recorded on the informationinputting sheet 29, and the access specifying pattern, provided by forexample the user's finger, are imaged by the imaging unit 11, theinformation for specifying the information in the information set,corresponding to the object, is matched to a position on the surface ofthe object, based on the imaging information, and the informationcorresponding to the object is automatically read out and recognized foridentifying the object. Alternatively, the pattern of the object ofrecognition, recorded on the information inputting sheet 29, and theaccess specifying pattern, provided by the user's finger, may bedirectly recorded on the object surface.

When the pattern of object recognition recorded on the object surfaceand the access specifying pattern provided by the user's finger areimaged by the imaging unit 11, the information for specifying theinformation in the information set, matched to the object, is matched tothe position on the object surface, based on the imaging information,and the information corresponding to the object is automatically readout and recognized for identifying the object, the object shape is notlimited to the sheet shape, such that an object identification code,specifically a Umark figure code 30, provided on the surface of acolumnar object 129A, as shown in FIG. 15A, or an object identificationcode, specifically a Umark figure code 30, provided on the surface of atriangular pyramid-shaped object 129B, as shown in FIG. 15B, may beimaged by the imaging unit 11 for recognition.

In the above-described information inputting and specifying device 10,the pattern of the object of recognition and the access specifyingpattern, provided by for example the user's finger, are imaged by theimaging unit 11, the information for specifying the information in theinformation set, corresponding to the object, is matched to a positionon the surface of the object, based on the imaging information, and theinformation corresponding to the object is automatically read out andrecognized for identifying the object. It is also possible not to usethe imaging unit 11 or the image recognition unit 12 and to read out theobject identification code, optically readably recorded on the objectsurface, by an optical information readout unit 81, to input theposition on the object to a sheet-shaped coordinate inputting unit 82,whereby the information for specifying the information in theinformation set corresponding to the object is matched to the positionon the object surface and the information corresponding to the objectmay be automatically read out and recognized for identifying the object,as in the information inputting and specifying device 10A shown in FIG.16.

This information inputting and specifying device 10A includes an opticalinformation readout unit 81, such as a scanner, provided with a CCDcamera or a CCD image sensor, for reading out the object identificationcode optically readably recorded on the object, such as theaforementioned information inputting sheet 29, that is the Umark figurecode 30.

The information inputting and specifying device 10A also includes asheet-shaped coordinate inputting unit 82, or a so-called touch panel,comprised of a matrix arrangement of a plurality of planar switcheswhich may be turned on as a result of e.g., pressure application, topermit detection of the position information of the pressed portion onthe sheet surface.

The sheet-shaped coordinate inputting unit 82 is formed of a sheet-likematerial, in which the electrical conductivity across its upper andlower surfaces is increased at the pressed area. The upper and lowersurfaces of the sheet-shaped coordinate inputting unit are provided withtwo sets of conductors arranged to cross one another. The conductor setsprovided on the upper and lower surfaces of the sheet-like material areconnected to voltage dividing resistors extending to cross therespective line directions. The output current is detected at both endsof the voltage dividing resistors.

The sheet-shaped coordinate inputting unit 82 is formed of a transparentmaterial or of a meshed material, so that, when the sheet-shapedcoordinate inputting unit is used as it is set on an object, such as theaforementioned information inputting sheet 29, the underlyinginformation inputting sheet 29 may be seen. It is noted that, if thesheet-shaped coordinate inputting unit 82 is opaque, the informationinputting sheet 29 is placed in use on the sheet-shaped coordinateinputting unit 82.

In the information inputting and specifying device 10A, employing thesheet-shaped coordinate inputting unit 82, the respective pagesaffording the schematic information formulated as a booklet or a catalogmay be formed by the information inputting sheet 29, and the schematicinformation may be correlated with the various detailed informationstored in the information system. In this case, such a system may beconstructed in which the book or catalog is first referenced as itspages are sequentially spread by way of giving a marking and theinformation inputting and specifying device 10A is overlapped with acorresponding page of the book or catalog to specify a necessaryposition on its surface to get to the detailed information. With thissystem, the operation of using the information stored in the informationsystem may be made in a state close to the state of everyday life, thusassuring smoother information inputting and specifying operations.

If the object identification code, that is the Umark figure code 30, isrecorded with a magnetic ink containing a high permeability material, amagnetic readout unit may be used in place of the optical informationreadout unit 81 in the information inputting and specifying device 10A.

An information readout unit for magnetically reading out the patternsformed by the magnetic ink, an electromagnetic unit 90, comprised of acore of a high permeability material, having a looped magnetic pathpartially cut away, and two coils 92A, 92B wound thereon. When theelectromagnetic unit 90, as the basic unit, has its partially removedportion of the magnetic path directed to the surface of the informationinputting sheet 29, having patterns recorded by the high permeabilitymagnetic material, a closed magnetic path is formed by a pattern of thehigh magnetic permeability material, forming the Umark figure code 30,such that the mutual inductance is changed across the two coils 92A,92B. Thus, by causing the current to flow through the coil 92A and bydetecting the electrical signals induced in the other coil 92B, it ispossible to detect whether or not the partially removed portion of thelooped magnetic path faces the pattern by the high magnetic permeabilitymaterial recorded on the information inputting sheet 29. The informationreadout unit 81A is so configured that a number of the electro-magneticunits 90, as the basic units, are arranged in a two-dimensional matricconfiguration, with the coils 92A being connected in series with oneanother in the vertical direction and with the coils 92B being connectedin series with one another in the horizontal direction, as shown in FIG.18.

With the information readout unit 81A, described above, the current iscaused to flow through the coils H1, H2, . . . , connected in serieswith one another in the horizontal direction, and the voltage induced inthe coils V1, V2, . . . , connected in series with one another in thevertical direction, is detected to read out the image of distribution ofthe pattern of the high magnetic permeability material without producingrelative movement with respect to the information recording surface.

Meanwhile, the information readout unit 81A is to be seen through topermit the underlying information inputting sheet 29 to be viewed.

1. An information inputting and specifying method comprising: storing ameaning and attribute of a letter in a storage means; reading out afigure code for the letter, indicating the meaning and attribute of theletter stored in said storage means, from a surface of an object;recognizing the meaning and attribute of the letter on the basis of thefigure code by use of figure code recognition means; recognizing theletter on the basis of the figure code by booting a letter processingprogram corresponding to the meaning and attribute of the letter by useof letter recognition means; and controlling an external device on thebasis of the result of recognition of the letter by use of controllingmeans, said figure code including a number of units, each unit having acircle mark and a plurality of rectangular marks arranged radiallyaround the respective circle mark.